Skip to content

[LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe #2231

Open
ApostaC wants to merge 2 commits into
mainfrom
lmcache-vllm-b200
Open

[LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe #2231
ApostaC wants to merge 2 commits into
mainfrom
lmcache-vllm-b200

Conversation

@ApostaC

@ApostaC ApostaC commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Description

Combines the tuned DeepSeek-V4 FP4 B200 vLLM AgentX recipe from #2224 / #2225 with the LMCache KV-offload backend from #2153, so the LMCache arm runs on the same image and serving flags as the vllm-simple and Mooncake arms and the backends are directly comparable.

Validated locally: bash -n passes; generate_sweep_configs.py full-sweep emits all 14 lmcache points (router attached on DEP points, total-cpu-dram-gb resolved from dram-utilization: 0.80); the changelog entry passes the Pydantic schema. Not yet run on hardware — needs a sweep run.

中文说明

#2224 / #2225 中调优后的 DeepSeek-V4 FP4 B200 vLLM AgentX 配方与 #2153 的 LMCache KV 卸载后端合并,使 LMCache 分支与 vllm-simple、Mooncake 分支在相同镜像和相同 serving 参数下运行,便于卸载后端间直接对比。

本地验证:bash -n 通过;generate_sweep_configs.py full-sweep 生成全部 14 个 lmcache 测试点(DEP 点带 router,total-cpu-dram-gbdram-utilization: 0.80 解析);changelog 条目通过 Pydantic schema 校验。尚未在硬件上运行——需要触发扫描。

Related Issue

Builds on #2224 / #2225 (tuned B200/B300 AgentX recipes) and #2153 (LMCache backend). / 基于 #2224 / #2225(B200/B300 AgentX 调优配方)与 #2153(LMCache 后端)。

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have tested my changes locally
  • I have updated documentation if necessary
  • If I changed a container image or config, I have already updated perf-changelog.yaml
    • New perf-changelog.yaml entries are appended to the end of the file (the file is chronological: oldest at top, newest at bottom)
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /reuse-sweep-run on this PR — do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps (remove and re-add the label to force one)

🤖 Generated with Claude Code

Combine the tuned B200 recipe from PR #2224/#2225 (nightly image, sparse
DSV4 FlashInfer attention, FULL_DECODE_ONLY CUDA graphs, AMXF4 mega-MoE
with the LMCache 0.5.1 KV-offload backend from PR #2153. The lmcache arm
drops --enable-cumem-allocator (cuMem/VMM allocations cannot be
CUDA-IPC-exported to the LMCache MP server) and runs otherwise identical
serving flags, so backends are directly comparable. Adds a standalone
dsv4-fp4-b200-vllm-agentic-lmcache config section mirroring the
vllm-simple concurrency ladder and a perf-changelog entry triggering it.

中文:将 PR #2224/#2225 的 B200 调优配方(nightly 镜像、稀疏 DSV4
FlashInfer 注意力、FULL_DECODE_ONLY CUDA graph、AMXF4 mega-MoE)与 PR
#2153 的 LMCache 0.5.1 KV 卸载后端合并。lmcache 分支仅去掉
--enable-cumem-allocator(cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache
MP server),其余 serving 参数与其他分支保持一致,便于卸载后端间直接对比。
新增独立的 dsv4-fp4-b200-vllm-agentic-lmcache 配置(测试点与 vllm-simple
阶梯对齐)及触发它的 perf-changelog 条目。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EOF
)
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

中文:将 perf-changelog 条目的 pr-link 填写为 PR #2231。

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ApostaC ApostaC changed the title [LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe / 在调优后的 DSV4 FP4 B200 vLLM AgentX 配方上新增 LMCache 分支 [LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe Jul 16, 2026
Comment on lines 328 to 350
--trust-remote-code
--kv-cache-dtype fp8
--block-size 256
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
--attention_config.use_fp4_indexer_cache=True
--max-model-len 1048576
--gpu-memory-utilization 0.92
--numa-bind
"${CUMEM_ARGS[@]}"
--no-enable-flashinfer-autotune
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--enable-prefix-caching
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}'
--no-disable-hybrid-kv-cache-manager
--disable-uvicorn-access-log
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
--max-num-seqs "$MAX_NUM_SEQS"
--max-cudagraph-capture-size "$MAX_NUM_SEQS"
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
"${FAST_MOE_ARGS[@]}"
"${OFFLOAD_ARGS[@]}"
)
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The rewritten VLLM_CMD in this file drops --tool-call-parser deepseek_v4 and --enable-auto-tool-choice, which were present in the pre-PR version of this exact script and are set in every other agentic vLLM recipe (dsv4_fp4_b300_vllm.sh, dsv4_fp4_mi355x_vllm.sh, dsv4_fp8_h200.sh, minimax/kimi variants). This looks like an accidental drop during the #2224 recipe rewrite rather than an intentional change, and since AgentX trace replay relies on tool_choice=auto, vLLM will reject those requests without --enable-auto-tool-choice, likely failing the sweep across all offload arms (none/vllm-simple/mooncake/lmcache).

Extended reasoning...

What changed: In the VLLM_CMD array construction (around lines 328-350 of benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh), the diff removes --tool-call-parser deepseek_v4 and --enable-auto-tool-choice while keeping the adjacent --reasoning-parser deepseek_v4 flag. The pre-PR version of this exact file had both flags (visible as removed lines in the diff), so this is not a case of the recipe never having them — they were explicitly dropped during the VLLM_CMD rewrite that pulled in the #2224 tuned recipe.

Why this is inconsistent with the rest of the repo: Every other agentic-coding vLLM recipe sets both flags together: dsv4_fp4_b300_vllm.sh (lines 209-210), dsv4_fp4_mi355x_vllm.sh (lines 396/398), and dsv4_fp8_h200.sh (lines 63-64) — confirmed directly by grepping those files. The minimax and kimi vLLM agentic recipes follow the same pattern. The PR description states the b200 script 'adopts the #2224 tuned recipe verbatim,' but #2224's own b300 sibling keeps both flags, so this omission is specific to the b200 rewrite and not an intentional nightly-vLLM behavior change.

Why nothing else in the script prevents this from mattering: The workload driven by build_replay_cmd/run_agentic_replay_and_write_outputs is the inferencex-agentx-mvp coding-agent trace replay, which is fundamentally tool-calling — requests are reconstructed with a tools schema and tool_choice. vLLM does not enable auto tool-choice/parsing by default; when a request specifies tool_choice=auto and the server wasn't started with --enable-auto-tool-choice (paired with a --tool-call-parser), vLLM returns an HTTP 400 ('auto tool choice requires --enable-auto-tool-choice and --tool-call-parser'). This is a request-level rejection, independent of whether the replay's default pre-canned mode discards live assistant responses — the request still needs to succeed for aiperf to count it as a valid data point.

Step-by-step proof of impact:

  1. The lmcache-arm (and every other arm: none/vllm-simple/mooncake) all share this single VLLM_CMD — none of the offload-backend case branches add tool-call flags.
  2. vLLM starts successfully (these flags don't affect startup), so wait_for_server_ready passes and the sweep proceeds to the replay phase.
  3. build_replay_cmd issues chat-completion requests for the AgentX coding-agent trace with a tools array and tool_choice=auto (required to reproduce realistic input-token counts for tool-augmented turns).
  4. Because --enable-auto-tool-choice/--tool-call-parser are absent, vLLM rejects essentially all such requests with a 400.
  5. aiperf's --failed-request-threshold (10%, per AIPERF_FAILED_REQUEST_THRESHOLD) is exceeded almost immediately, failing the run for all 14 lmcache sweep points plus the parent vllm-simple/mooncake points that share the same script.

Fix: Re-add --tool-call-parser deepseek_v4 and --enable-auto-tool-choice to the shared VLLM_CMD array (near --reasoning-parser deepseek_v4), matching the pre-PR file and every sibling recipe.

Comment on lines +285 to +289
unset VLLM_USE_SIMPLE_KV_OFFLOAD
OFFLOAD_ARGS=(
--kv-transfer-config
"{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new lmcache arm's --kv-transfer-config sets kv_connector":"LMCacheMPConnector" but omits kv_connector_module_path, which both sibling scripts wiring the same connector (kimik2.5_fp4_b200.sh:150, dsv4_fp4_mi355x_vllm.sh:344) include. LMCacheMPConnector is an out-of-tree class that vLLM can only resolve via kv_connector_module_path + kv_connector; without it, vllm serve will fail to construct the KV connector at startup, so every one of the 14 lmcache sweep points will crash before producing a result. Fix by adding "kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector" to the JSON at line 288.

Extended reasoning...

The bug: The new lmcache case in dsv4_fp4_b200_vllm.sh (lines 285-289) builds OFFLOAD_ARGS as:

"{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{...}}"

This is missing the kv_connector_module_path key. LMCacheMPConnector is shipped by the lmcache pip package (installed a few lines earlier via agentic_pip_install ... lmcache==$LMCACHE_VERSION), not by vLLM itself. vLLM'''s built-in KVConnectorFactory only knows connector classes it ships in-tree by bare name; to resolve an out-of-tree connector class, it needs to dynamically import the module named by kv_connector_module_path and then look up kv_connector as a class name inside that module. Without the module path, vLLM falls back to its built-in name registry, which has no entry for LMCacheMPConnector, and connector construction fails at vllm serve startup.

Why the existing sanity check doesn'''t catch it: The script runs python3 -c \"import lmcache.integration.vllm.lmcache_mp_connector\" >/dev/null a few lines before launching the server. That only proves the module is importable in a throwaway Python process — it does not register the connector with vLLM'''s factory in the actual vllm serve process, so it provides no protection against this specific misconfiguration.

Cross-script evidence: Both other scripts in this repo that wire up the identical LMCacheMPConnector include the field explicitly and consistently:

  • benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:150: \"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",...
  • benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh:344: same field, same value, otherwise near byte-for-byte the same JSON shape (same lmcache.mp.host/lmcache.mp.port extra_config keys).

The new B200 arm is the only place in the codebase wiring LMCacheMPConnector without this field, strongly suggesting it was simply dropped rather than being some new supported invocation.

Impact: This is not a subtle perf regression — it'''s a hard startup crash. Every one of the 14 lmcache sweep points defined in configs/nvidia-master.yaml (dsv4-fp4-b200-vllm-agentic-lmcache: 3 TP8 points + 11 TP8-DEP8 points) launches vllm serve with this malformed --kv-transfer-config, so the server process would fail while constructing the KV connector, before any benchmark traffic is served. This defeats the entire purpose of the PR, which is to add this lmcache arm for comparison against vllm-simple and Mooncake.

Step-by-step proof:

  1. KV_OFFLOAD_BACKEND=lmcache is set for a sweep point; the script enters the lmcache) case at line ~230.
  2. The LMCache MP server starts and becomes healthy (this part is unaffected — it'''s a separate process from vLLM).
  3. OFFLOAD_ARGS is built at line 285-289 as --kv-transfer-config '\''{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",...}'\'' — no kv_connector_module_path.
  4. vllm serve is launched with this flag (line ~372, via \"${OFFLOAD_ARGS[@]}\").
  5. vLLM parses --kv-transfer-config, sees kv_connector=\"LMCacheMPConnector\" with no module path, and attempts to resolve it against its built-in registry (which contains e.g. LMCacheConnectorV1 but not LMCacheMPConnector).
  6. Resolution fails, and vllm serve exits/crashes during connector construction, before the health-check endpoint ever comes up — wait_for_server_ready at the bottom of the script will time out and the whole benchmark point fails.

Why this wasn'''t caught: The PR description explicitly says validation was only bash -n (syntax check) and generate_sweep_configs.py full-sweep (config generation), neither of which launches an actual vLLM server, so this startup failure was never exercised.

Fix: add \"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\" to the JSON at line 288, matching the sibling scripts exactly.

@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentx AgentX benchmarks, recipes, and infrastructure full-sweep-enabled

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant